Learning from Examples and Side Information

We set up a theoretical framework for learning from examples and side information which enables us to compute the tradeoo between the sample complexity and information complexity for learning a target function in a Sobolev functional class F. We use the notion of the n th minimal radius of information of Traub et. al. 23] and combine it with VC-theory to deene a new quantity I n;d (F) which measures the minimal approximation error of a target g 2 F by the family of function classes with pseudo-dimension d under a given side information which consists of any n measurements on the target function g constrained to being linear operators. By obtaining almost tight upper and lower bounds on I n;d (F) we nd an information operator ^ Nn which yields a worst-case error no larger than a logarithmic factor in n and d than the lower bound on I n;d (F). Hence to within a logarithmic factor it is the most eecient way of providing side information about a target g under the constraint that the information operator must be linear and that the approximating class has pseudo-dimension d.